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ABSTRACT 

This evaluation guide resulted from a 2-year project 
that conducted literacy audits at 10 manufacturing companies, 
assessed 3,291 v^rkers, developed customized assessments and 
curricula, and provided 104 courses to 948 participants in the 
Chicago area. The guide is a hands-on resource to assist companies, 
unions, or educational agencies in setting up a system for measuring 
the effectiveness of a basic skills program. Using examples from the 
project, the guide covers the following topics: (1) the need for 
basic skills in the manufacturing industry; (2) issues in evaluating 
basic skills pro 6 rams; (3) program evaluation; and (4) the four 
levels of program evaluation (reaction, learning, transfer of 
learning, and organizational impact). Samples for conducting each 
level of program evaluation are provided. Contains 11 references. 
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Introduction 



Project Overview 

"Workplace literacy in a Total Quality Management Environment for the Manufacturing 
Industry" was funded by the U.S. Department of Education to provide workplace literacy 
instruction to ten manufacturers in the Chicago area. It was in operation from the period May 
20, 1992 - May 31, 1994. During that time, it conducted literacy audits at all ten companies, 
assessed 3,291 workers, developed customized assessments and curricula, and provided 104 
courses to 948 participants. 

Profile of Manufacturing Partners 

The ten participating companies ranged in size from 200 employees to 1 500. About half of the 
companies were unionized. All were involved in some form of Total Quality Management 
initiative. The most frequent quality strategies were adoption of Statistical Process Control 
(SPC), re-organization into work teams, and employee involvement. 

Profile of Employees 

The employees who participated in the program were 45% male and 55% female. The largest 
group of workers were Hispanic (49%), followed by White (28%), Asian (14%) and Black (8%). 
The majority had been employed by the company between 1 - 5 years. Approximately 30% had 
been employed more than 1 1 years. 

Profile of Services 

Of the 104 courses, 33 were ESL, 28 were reading/writing, 35 were math, 2 were oral 
communication for customer interaction, and 6 were GED. All courses were customized to the 
needs of the individual company and the curriculum was competency-based, functional 
context. Average class size ranged between 7-10 employees. Ninety percent of the companies 
offered the courses on either 50% or 100% work time. 

Purpose of this Guide 

This guide is written from the perspective of our experiences as basic skills providers. 
Our intended audience is the practitioner, be it company, union or educational agency, who is 
looking for a hands-on resource to assist in setting up a system for measuring the effectiveness 
of a basic skills program. Our discussion in this guide reflects our actual experience which is 
limited to the manufacturing industry and to functional context, basic skills programs. We have 
incorporated examples of results and our actual tools wherever possible. Because we feel thrit 
experience is cumulative and results from other projects drove our evaluation design in this 
project, we referenced other companies' experiences as well as the experiences from this 
particular project. 
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Observations about basic skills programs 

Having been involved with planning and implementing basic skills programs at more than 100 
manufacturing companies, we have made five key observations about the environment under 
which companies and the basic skills programs must operate. 

1. The basic skills of the workforce are critical to the ability of the manufacturing 
industry to transform itself into "world-class manufacturers.** 

This implies there is an immediate need for basic skills programs whose outcomes are tied to 
the transformation to world<:lass manufacturers in as short a time as possible. 

2. Manufacturing companies have limited resources and time to successfully make 
this transformation. 

The transformation often involves the purchase and use of expensive technology, extended 
training in technical areas which requires a commitment of time off the floor as well as expense, 
development and adoption of quality standards and tools for measurement, and a philosophy 
of quality which is disseminated throughout the organization. No one company has sufficient 
people, time, and financial resources to accomplish all these objectives in the short-term. 
Priorities must be set, and in setting those priorities, basic skills programs must compete for the 
company's limited resources, as well. 

3. Manufacturers operate in a very dynamic environment. 

There are continual and often unpredictable changes in the nature of the business, technology, 
competition, products, production flow, work orders, and the workforce. Such changes 
require that a basic skills program is flexible. 

4. Basic skills programs are not an entity unto themselves - rather they are an 
integral part of the entire company and penetrate all the company's functions. 

Basic skill programs are affected by all the dynamic events which take place. If a company is 
downsizing, program attendance will be affected. If overtime is required to complete orders, 
attendance will likely decline and the program may be temporarily suspended. If pay for 
knowledge/skills is going to be implemented, the desired level of basic skills may become one 
of the benchmarks. New hiring policies may reflect the desired level of basic skills as indicated 
by the literacy audits. Implementation of SPC in the near future will make math assessment and 
training a priority. Successful participation and completion of the program may become a part 
of the performance appraisal and promotion processes. These are but a few examples of how 
the basic skill program interfaces with the company. 
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5. Evaluation permeates all business operations as the management must decide 
in which efforts to place its limited resources. 

The goal of the company is to make a profit and to allocate that profit in such a manner as to 
continue the existence and survival of that organization. As such, there is competition for all 
resources. While "basic skills programs" are a "nice thing to do", they must, by their very nature 
and by the nature of the organization, p*ove their value against the value which could be 
provided by other initiatives. 
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The Need For Basic Skills 
in the Manufacturing Industry 



Global competition hi price, on-time delivery, and improved quality have forced the American 
manufacturing industry to change dramatically by adopting a Total Quality Management (TQM) 
approach. Expectations of production workers have increased as the demand for high 
productivity, zero defects and new technology impacts the factory floor. Workers are being 
asked to contribute ideas; to learn new skills such as Statistical Process Control (SPC), cellular 
manufacturing, and geometric tolerancing; to actively participate in team production and joint 
problem-solving; and to become cross-trained in a variety of jobs within their respective 
departments. In order to effect these changes, the workforce MUST have the basic math, 
reading, writing, and language skills. 

Successful adoption of TQM depends on a number of factors, but company-wide training of the 
workforce is one of the most important factors. As a manufacturer changes its approach from 
traditional manufacturing to a world class company, it is the basic skills level of the workforce 
that will determine whether or not that transition will be successful. The chart on the next two 
pages summarizes the current manufacturing practices of today and sample basic skills that are 
necessary for attaining the practices of the future. 
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Basic Skills Within Manufacturing 



Manufacturing ^ Sample Manufacturing of 
Today Basic Skills Tomorrow with TQM 


Problem-solving and 
decision-mating that is limited 
to management. 


Read flowcharts. 
Think critically. 
Read and construct fishbone 
cause-effect charts. 
Read and construct check sheets. 
Write descriptions of problems 
and possible solutions. 


Team problem-solving and 
decision-making that is driven 
throughout the organization. 


Flow manufacturing. 


Read individual customer spec sheets. 
Read production charts and 
directions which change daily. 
Maintain job cards and records 
of production. 
Fill out forms for requisitioning raw 
materials for customized orders. 


Just-in-Time manufacturing. 
Materials Resource Planning. 


One person-one job. 


Oral communication skills to 
participate in training. 
Read training materials. 

Complete tests/assessments. 


Cross-training— skilled in a 

variety of jobs. 
Manufacturing work cells. 


Little or no employee involvement 
in the management of the company. 


Orally participate in meetings. 
Read agendas and meeting notes. 
Take notes in meetings. 
Complete survey forms. 
Complete process forms. 


Employee involvement throughout 
the organization. 


Working directly under a supervisor. 


Oral communication skills such as 
giving instructions, giving advice, 
solving problems, negotiating, giving 
positive or constructive feedback, etc. 


Working in teams. 


Low technology. 


Read technical manuals. 
Read computerized directions. 
Oral communication skills to 
participate in technology training. 


High technology. 
Sophisticated equipment. 
Use of computers. 


Reliance on inspection department 
for quality. 

-V 

1 


Calculate production and rejects. 
Fill out charts. 
Analyze data. 
Measure in metric. 
Calculate with whole numbers, 
decimals and fractions. 
Write summaries of trends and ^ents. 


Employees who inspect their own work, 
use Statistical Process Control and 
the implementation of other 
quality strategies. 
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Manufacturing . Sample Manufacturing of 
Today ■ U^' Basic Skills Tomorrow with TQM 


No certification— eaon company has a 
variety of suppliers and customers with 
little or no demands for formalized 
quality procedures. 


Orally describe job processes. 
Orally describe purpose and 
nature of company. 
Read flowcharts. 
Read work procedures. 
Write work procedures. 


Quality audits. 
Supplier certification. 
ISO 9000. 


Training limited to middle and 
upper management. 


Read training manuals and materials. 
Listen and take notes. 
Ask and answer questions. 
Write in-training exercises. 
Take tests. 
Give oral and written feedback on 
tne success of the training. 
Participate in problem-solving. 


Training is essential for all workers. 


Limited or no interaction 
with customers. 


Oral communication skills to request 
feedback, understand suggestions, etc. 
Read written reports and customer 
survey responses. 
Correspond with customers. 


Both internal and external 
customer interaction. 


Pay for seniority or by job category. 


Oral communication skills to 
participate in training. 

ncaO U all llliy IllalcJ lalo. 

Take notes. 
Complete tests/assessments. 


Pay for skills and knowledge. 


Primarily company-initiated 
safety procedures. 


Read Material Safety Data Sheets. 
Read safety bulletins. 
Fill out safety reports. 
Orally participate in safety training. 
Read safety training materials. 
Read government documents. 
Participate in safety committees. 


Increased government regulations 
and formal procedures for safety 
and disposal of hazardous materials. 
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The road to world-class manufacturing is long and the trip is expense Because of the broad 
nature of the various initiatives needed to be successfully undertaken, a company must carefully 
plan its route. Expediency is critical; as is an integrated, coordinated, efficient effort to 
undertake all the initiatives. 

Evaluation of the progress and success of each initiative is necessary to validate efficiency and 
expediency. As such, it is the purpose of this guide to share experiences and explore best 
practices leading to the design of a practical approach to measuring the effectiveness of basic 
skills programs in the manufacturing industry. 
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Issues in Evaluating Basic Skills Programs 



There are numerous issues facing a business/education provider partnership in evaluating the 
success of the basic skills program. Listed belcw are nine of the most common issues which 
the project has faced. 

1. Different goals of a basic skills program. 

In the experience of the project, there have been three distinct goals for the programs: 1) to 
enhance and accelerate the successful implementation of new initiatives, 2) to improve 
productivity in the short-term, and 3) to enrich and expand the overall educational level of the 
workers for both the workplace and for their personal lives. While a program may have multiple 
goals, at least one of the above is the primary goal. It is important to identify the goals because 
each goal will drive the design of the evaluation. 

The first goal - to enhance and accelerate the successful implementation of new initiatives - has 
been the primary goal for the majority of our partnering companies. In this project, the most 
frequent initiatives have been the development of a Total Quality Management (TQM) system, 
adoption of Statistical Process Control (SPC), reorganization into work teams, and continuous 
process improvement. Measuring the impact of the basic skills program in adopting new 
initiatives is difficult because there is no baseline data. 

The second goal - to improve productivity in the short-term - implies that there is a measurable 
problem which basic skills training can solve. If this is a major problem, there are likely to be 
a variety of interventions taking place because of the urgency of the problem. For example, 
a safety problem exists because the workforce lacks the English to comprehend the safety rules. 
To resolve the problem the company can translate the rules into the workers' native language 
or provide safety training in the workers 1 native languages. These interventions can take place 
sooner than providing English training to all workers. If all three interventions occur and the 
accident/injury rate decreases, which intervention gets the credit? If the program is voluntary, 
it is even more difficult to attribute the decline in the injury rate to the presence of the basic skills 
program. 

Having worked with more than 100 companies in a five-year period, there exists only one 
instance where there is a concrete example of improving productivity in the short-term. In this 
instance, a customized assessment was developed and a minimum score determined. The 
assessment was made mandatory, as was participation in the program until the designated score 
was attained. Since no other changes took place at the worksite during the 6-month program, 
positive changes in scrap and waste were attributed to the program. 
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2. Changing goaj£. 



Goals for the program fttfy change during the project. This affects th^m^unt of available data. 

♦ Company A stated \$ a voluntary, enrichment program. As manag^e^t began to recognize 
the need for btyic Sjcills as a means for successful implementati<A o{ new initiatives, the 
program becac^s mandatory. This change took place in the fux^l months of the project. 
Therefore, tim^Va^ not available to collect enough data to demot^rat^ organizational impact 

3. Changing co^at^y factors. 

A variety of factory <Vithin the company affect the presence as well ztf th^ potential success of 
the basic skills prc^taAv A few of these factors are changes in ma*#ge^ent, changes in the 
actual workforce Oyolfs or hiring), upturns or downturns in production, customer demands, 
or new initiatives, foch can affect the goal and structure of the basi^ skills program as well as 
the evaluation des^. *fhe following are examples of how changing totA^any factors affected 
the basic skills prQ^taAi during an eighteen-month period: 

♦ Company A m^j^g^flient made a commitment to basic skills trailing £nd strongly encour- 
aged assessme^/p^rticipation with 100% release time. Projected ^mb^r of courses was 12. 

Company-wide ^s^sment was completed and two courses wety offend. A new produc- 
tion mans er halted ^11 training for more than 6 months. When an^tHe^ production manager 
took over, all tuMntog was re-initiated, and more than 14 course^ tye*£ offered in the next 
12 months. 

♦ Company B ma^ge^ient made a commitment to basic skills trair^Ag. Participation in both 
the assessment find training was voluntary. All courses were otf£re<3 on work-time. The 
company bega^ <ih^pgiag from a single factory to four focus-fact^es, ISO 9000 was being 
applied for, an<J a f^y for skills program was being designed. Tflt company reduced the 
release time to Later the company reconsidered the value Qk(\h§ basic skills program. 
After the three feitUtives were achieved, the company, in disc^siOfi with the provider, 
determined th^f assessment and participation in training wou^ b£ mandatory and that 
training would held on 100% work-time. 

+ At the time of application for Federal funds, Company C was ^m^itted to a basic skills 
program with $0% please time. During the nine month waiting ^et^cd before beginning 
the program, b\|Sin^£s took a downturn and there were some cha *jge$ in management. One 
course was ofl^fed Put on voluntary time. A severe thundersto^ then destroyed a large 
portion of the j^ct^jy and major repairs were a priority. Participation dropped and the 
program was d^co^tinued. 

4. Absence of d^t^* 

While all company have some productivity data, the data maf he insufficient to track. 
Alternatively, conv^ni^s may be expanding or changing their data collection system and it will 
be difficult to tracjtf th^ data during the time of the basic skills prog^m- 
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5. Inability to track the data to the program. 

When data is available, it may not be possible to track the data to the individuals participating 
in the program. Data regarding productivity or personnel measures is usually company-based 
or department-based. In the situation of work teams, the data is usually team-based. If 
participation in the basic skills program is voluntary, it is difficult to track and attribute any 
changes in the data to the success of the individuals in the company. One example is: 

♦ In Company A, participation in the program was voluntary. All data regarding productivity 
was team-based. No teams had significant participation in the program to attribute produc- 
tivity changes to the program. 

6. Numerous factors affecting any changes. 

In as much as manufacturing companies are very dynamic, many changes can take place at the 
same time. 

♦ Company A began an SPC program to satisfy one of their major customers. Math classes 
began after a skill gap was identified. A month later, a gainsharing program was announced, 
and with a 60% non-English-speaking workforce, it was decided that English classes would 
be important. Within the next four months, the company added capital equipment while 
introducing a new product, and lost nearly 20% of the business from the customer that 
requested SPC from its suppliers. 

7. Lack of consensus on the need or value of the program. 

Support for the program must be company-wide. If management supports the program but 
supervisors do not, workers may not be encouraged to attend or even released to attend classes. 
Support for the program must also come from the workforce itself. If attending class increases 
the workload of the nonparticipating workers, the support and encouragement for those 
participating in the program may not be there. 

8. Nature of the program - voluntary vs. mandatory. 

In voluntary programs, those who come forth to participate may not be those workers who need 
the training the most. Hence, changes in productivity may not be noticeable. On the other hand, 
while all workers with minimal skills may be mandatorily sent to training, a poor attitude may 
prevent the workers from learning or even discourage others from learning. 

9- Limited time & resources. 

A company may not be able to afford sufficient release time for noticeable organizational impact. 

♦ Company A has a workforce of 2500. Management has determined that the maximum 
number of basic skill classes that production can afford during the year is eighteen and that 
classes have to be spread over several departments and shifts. Because the training is spread 
over the entire company, impact is impossible to measure during the first year. 
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Program Evaluation 



The focus of this guide is on the evaluation of a functional context program, that is, the 
determination of how well the entire program is functioning. The project has discovered that 
the most useful framework for evaluating programs is Kirkpatrick's four-level hierarchy which 
was also referenced in the U.S. Departments of Education and Labor publication The Bottom 
Line: Basic Skills In The Workplace . These four levels consist of 1) participant reaction, 2) 
participant learning, 3) participant performance, and 4) organizational results. 

The first level, participant reaction, measures the participants' perceptions about the course 
as well as the program. Typical tools for measuring this level are what the field commonly calls 
"smile sheets" - the five point, likert-scale rating questionnaires, and open discussion or 
interviews with the participants. 

The second level, participant learning, measures the amount of learning which has taken 
place during the course or during the program. The most commonly used tool is the pre/post 
assessment. 

Level three and four become more complicated in the hierarchy. The third level, participant 
performance, attempts to measure the transfer of learning from the course to the actual 
workplace. This level is critical because of the old adage, "You can lead a horse to water, but 
you can't make it drink." While a program can teach knowledge and skills, the question is 
whether or not that knowledge or these skills will be applied to the job. If a worker has 
demonstrated that he or she has learned to read a job aid, will that worker then refer to that job 
aid on the job, or will that worker continue to ask a co-worker about the information contained 
in that job aid? It is this third level which attempts to document the answer, to the question of 
the application of those skills. Common tools for this level include supervisors 1 interviews or 
questionnaires, job observation with performance checklists, and self-appraisal. 

Level four, organizational impact, presents the biggest challenge to basic skills programs. 
This level focuses on the impact of the basic skills program on the company. The success of 
measuring the impact depend on four major factors: 1) the presence of quantifiable data, 2) the 
access to such data, 3) a solid relationship between the data and the basic skills program, and 
4) an absence of other factors which could also affect the data. Examples of data include 
production data, such as productivity, scrap, customer complaints, cost, and quality measures 
and workforce data such as performance reviews, accidents/injuries, promotions, and absen- 
teeism. 

The chart below summarizes these four levels. Further discussion on the purpose, advantages 
and disadvantages, and actual tools used by the project can be found in the next four chapters. 
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Four Levels of Evaluation 









REACTION 


Obtain learner's input on 
the content, approach, and 
outcomes of the instruction 


+ "1-5" questionnaires 
+ Open-ended question- 
naires 

+ One-on-one interviews 
+ Focus groups 


LEARNING 


Measure the amount of 
learning 


+ Pre-post tests 

+ Competency checklists 


TRANSFER OF 
LEARNING 


Measure the learning which 
has been transferred to the 
job 


+ Performance checklists 
+ Surveys - workers, su- 
pervisors, managers 
+ Interviews - workers, 
supervisors, managers 


ORGANIZATIONAL 
IMPACT 


Measure the program 
results on the company 


+ Production data: 
complaints 
downtime 
errors 

productivity 
+ Workforce data: 

absenteeism/tardiness 
grievances 

promotions 
suggestions 
requests for assistance 
with personnel issues 
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Level 1: Reaction 



If a basic skills effort is to be sustained in any company or organization over a long period of time, 
then it must have an effective evaluation component. a How much value has it added to the 
organization?" is the critical question. 

It is well accepted that at the first level the most frequently used method of evaluation is 
participant reaction. Instruments that were used to measure participant reaction asked the 
fundamental question, "How did you like the basic skills program?" It is probably true that many 
organizations will not continue a basic skills initiative over a long period of time without 
demonstrating transfer and impact. The case for direct transfer and most certainly impact is 
difficult to establish for basic skills programs especially in the short term. Reaction evaluations, 
however, can get the program a over the hump." In their own right, and as has been the case 
in the companies in this project, participant reaction (or first level evaluations) are as important 
as, if not more important than, any of the other three types of evaluation, especially in the 
infancy of a program. 

First level evaluations can have several purposes, and therefore several outcomes. First and 
foremost: they are indicators of customer satisfaction. Whether administered concurrently or 
after the completion of a course or program, they are a barometer of participant/organizational 
satisfaction. As a result of our experience in this project, it is our recommendation that, 
minimally, basic skills courses be evaluated on this level during, as well as after, the course is 
completed. 

Participant surveys are the most frequently used tools to measure reaction. Since the purpose 
of the surveys is to obtain input and feedback on the appropriateness of the instructional 
activities and pace, the clarity and usefulness of the materials, the instructor's performance, and 
the value of the lessons, the surveys can be very useful in indicating a need for corrective action. 

Focus groups are an additional tool that can be employed with surveys on first level 
evaluations. Several companies have utilized this method with great success. After the paper 
and pencil surveys were completed, the training manager conducted a focus group with the 
class covering the topical areas in the survey. Students who do not express themselves well in 
writing had an opportunity to share opinions orally. With the instructor absent, participants 
were also much freer with constructive comments as there was no fear of hurting anyone's 
feelings. 

The value of first level evaluation data is that it provides corrective action opportunities. Until 
organizational impact can be determined and data made available, participant surveys are able 
to provide end user feedback that can sustain the basic skills initiative. 



O L- Mrowlckl / J. Conrath — The Center - Resources for Education and Northern Illinois University - Business and Industry Services 

ERIC ■»• 



The following chart summarizes the strengths and weaknesses of each tool. 

Tools for Measuring Reactions 




Mid-term 

participant 

survey 



+ Easy to administer 
and compile 



+ Forms can 
customized 



be 



+ Corrective action 
can take place before 
the course has ended 

♦ Measuring the trans- 
fer of learning can 
begin 

+ Anonmyous 
responses 



+ Tendency for the "halo 
effect" 

+ Inadequate basic skills to 
complete the form 

+ Subjective ratings 

+ Reluctance on the part 
of some participants to 
give feedback 



+ Participants have the basic 
skills to complete the forms 

+ Data can be easily collected 
and quickly compiled 



End of course 
participant 
survey 



+ Easy to administer 
and compile 

+ Forms can be 
customized 

♦ Measuring the trans- 
fer of learning can 
begin 

+ Anonmyous 
responses 



+ Too late for corrective 
action 

♦ Tendency for "halo 
effect" 

♦ Inadequate basic skills to 
complete the form 

♦ Subjective ratings 

♦ Reluctance on the part 
of some participants to 
give feedback 



♦ Participants have the basic 
skills to complete the forms 

♦ Data can be easily collected 
and quickly compiled 



Focus Groups ♦ Verbal responses 

♦ Participants can 
build on others com- 
ments 



Not anonymous; some 
participants may be 
reluctant 

Participants need oral 
communication skills 



♦ Participants have the oral 
communication skills to par- 
ticipate 

+ Participants arc open and 
willing to share perceptions 

♦ A skilled facilitator is 
available 
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Project recommendations: 

While the project recognizes that each company is unique and a customized evaluation plan 
should be created for each company, certain tools and strategies have proven useful. In the 
experience of the project, we recommend the following: 



1. Mid-course participant surveys should be utilized if time permits. (See Exhibit 1 for a 
sample.) 

2. End-ofcourse participant surveys should be utilized. (See Exhibit 2 for an example which 
is appropriate for a beginning level of basic skills and Exhibit 3 for an example for 
participants with higher basic skills.) 

3. Focus groups at the end cf the course are very beneficial. (See Exhibit 4 for an example 
of participant feedback which resulted from the focus groups.) 
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I Workplace ESL Mid-Term Training Survey 

Company Name: Date: 



Exhibit 1 



Instructor s Name: Course: 



We would like to know how you feel about the class at this time. Your answers to these questions 
will help us make improvement to the class. 

Yes No 

1. Will the things you arc learning be useful on the job? 

2. Docs the instructor give good examples? 

3- Docs the instructor help you if you have a question? 

4. Do you like the books and handouts that you are using? 

5. Do you spend enough time speaking English in class? 

6. Do you spend enough time reading English in class? 

7. Do you spend enough time writing English in class? 

8. Arc you becoming more comfortable using English now? 

9- Has your use of English increased on the job? Explain. 



10. Does your supervisor support your efforts to speak and 
read English better? Explain. 



11. Has your use of English increased outside of work? Explain. 



1 2. Circle one then explain. Is this class ... Too easy? Too difficult? 

13. What do you like most about the class? 

14. What can the instructor do to help you learn more? 

15. Do you have any additional comments? 
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COURSE EVALUATION FORM 



Exhibit 2 



Company/Course: 

Instructor 

Schedule: 



1. The course was: 



Very good Good OK Poor Very poor 



2. The materials were: Very good Good 



OK 



Poor Very poor 



3. The instructor was: Very good Good OK Poor Very poor 



4. My English in my job: Improved 

a lot 



Improved 
a little 



Did not 
improve 



5 Would you like to take another course? 



Yes 



No 



6. Would you recommend this course to a co-worker? Yes 



No 



1. What did you like about the course? 



8. What didn't you like about the course? 



9. What would you add to this course? 
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Exhibit 3 



COMPANY'S NAME 



INSTRUCTOR'S NAME 



TRAINING FEEDBACK SURVEY 

TODAY'S DATE. 

COURSE 



Thank you for participation in fhfs training program. Please help 
make this course even better by sharing your ideas with us. 



PA RT A- fi-RNFRA f HflA f TTV 






D I RECTI ONS: Please circle yoar responses to tlie folio wing questions. 






1. Did the things that you learned in this class help you on the job? 


YES 


NO 


2. Did instructor give good, clear explanations? 


YES 


NO 


3. Did the instructor know the course material? 


YES 


NO 


4. Did the instructor give you help when you needed it? 


YES 


NO 


5, Did the instructor make you feel comfortable and give you confidence? 


YES 


NO 


6. Did the class usually start and end on time? 


YES 


NO 


7. Was the class interesting? 


YES 


NO 


8. Would you recommend this ciassto a co-worker? 


YES 


NO 


9. Did your supervisor support your participation in this class? 


YES 


NO 


10. Would you like to lake another course? 


YES ' 


NO 



PARTE: RATED RESPONSES 

DIRECTIONS: Please circle the response that best matches yoar opinion. 
11. Overall, this class was 



12. The materials used were 

13. The instructor was 

The material presented was 



ERLC 



VERY 
GOOD 

.YERY 
GOOD 

VERY 
GOOD 



GOOD 



GOOD 



GOOD 



TOO EASY 



OX 



OJC 



O.K. 



JUST RICH* 



POOR 



POOR 



POOR 



TOO HARD 



VERY 
POOR 

VERY 
POOR 

VERY 
POOR 
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PARTC: DETAILED RESPONSES 

DIRECTIONS: Please answer tlic following questions. Take time to titinkaboat yoar answers. 
Don 't worry about yoar spelling or grammar — we need yoar ideas . 



15. What did you like the most about this class? 



16. What did you like the least about this class? 



17. What can you do now tnat you couldn't do before you took this class? 



18. What changes would you suggest to make this class even better? 



ERLC 



THANK YOU FOR YOUR PARTICIPATION! 
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Exhibit 4 



Focus Group Summary 

Listed below is a summary of actual comments from focus group discussions at one company. 



Figure 4- Participant Responses about Goals Met 



Better communication now. 
First time I wrote English. 
I can understand more English and 

conversation. 
I am able to slate a problem, explain to 

supervisor and ask questions about 

safety. 

can understand department meetings, 
know what to say at meetings. 
My vocabulary is better. 
I know how to communicate with my 

supervisor and fellow workers. 
I am not afraid to speak English to my 

supervisor. 
I understand my job and meetings 

better and safety prevention. 
I know how to communicate about 

accidents and prevent accidents. 
I understand safety prevention- example 

why we use safety glasses. 
I am able to ask questions about safety. 
I know how to take minutes. 
I made my first CIP [suggestion] by 

myself and will write more. 
Now I know how to write the CIP 
[company suggestion form] I have 
been working on. 
I have written a CIP. 
I am learning to trust myself. Next time 

I won't be afraid to write English. 
I know the difference between past, 

present and future tenses. 
Now I can read a traveler [job ticket] 
very well; where to go, what to look 
for and what I need. 



At meetings I used to just listen, now 
I speak up and ask questions and 
give ideas. 
I have a seven year old daughter. I was 
afraid to help her with her school 
work and now she tries to help me. 
If I don't see my kids at home, now 

writing them notes. 
The teacher helped me talk and 

understand. 
I can tell about my job and I under- 
stand what other guys do in their 
department. 
First tr e I came to class I know 
nothing; now I know a lot of words 
and about safety. Very nice. 
I can use English and fill out an 

application. 
Had 200 hours at an adult learning 
center. Now I know the difference. 
There I learned to read words. That 
wasn't reading. You're really 
teaching me to read. 
Can read procedure sheets. 
Can read instructions for pull aways 

[job procedure]. 
Know how to read and fill out a scrap 
card faster and don't have to ask as 
many questions. 
Can explain scrap cards to others now. 
Know how to read bar graphs and 

understand the goal line. 
Can read instructions better. 
Supervisor comment: Employee is 
learning about computers and is 
able to train others. 




25 



Level 2: Learning 



The second level of evaluation attempts to measure the amount of learning that has occurred. 
Measuring what is taught in the classroom focuses on the content as well as the participants 1 
progress. Again the issue is value. 

The measurement of learning is undergoing its own sort of evaluation. The state-of-the-art 
workplace literacy programs are currently conducted within a functional context. Whether 
stated formally or not, courses are structured around competencies. How to measure the 
competency is one challenge, and how to measure participants' progress to a company that 
expects high value is quite another. The objective is to be accountable though accurate 
measurement and reporting. It is important that we, as educational providers, set realistic 
expectations, conduct training that focuses on competencies, and report results that describe 
a level of performance in behavior while maintaining confidentiality in order to demonstrate 
high value to the company. 

Following is a discussion of measuring and reporting participant learning with three tools: pre/ 
post assessments/percent improvement, module completion, and competency checklists. 



Pre- and Post-Tests and Percent Improvement 

Comparing pre,test and post,test results is used to answer the basic question, "Has learning 
taken place?" Measurement at this level is straightforward and based on empirical data. 

Before training begins, an assessment is given (either a standardized tool such as the Test of 
Adult Basic Education (TABE), or a customized test) to measure participants' performance in a 
general area of knowledge. Training recommendations are made based on test results; however, 
the scores themselves remain confidential and are only revealed to each individual participant. 
The course content of each training session varies, according to the group's and participants' 
needs. Instruction is not geared to "teach to the test"; rather, course content is general in nature, 
with an emphasis put on workplace materials. 

At the end of instruction, the same assessment (used as the pre-test) is given and the results are 
compared. The company is informed of the percent of improvement for each participant, but 
again, no scores are reported. Table 1 below shows how the Percent Improvement is calculated. 
An example of a customized assessment is presented in Exhibit 6. Table 2 provides a sample 
company report of Percent Improvement. 
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Table 1 

Calculating Percent Improvement 

Formula (Post-test Score - Pre-test Score) / Pre-test Score 



Examples 

Participant A Pre-test Score: 25 out of 30 

Post-test Score: 30 out of 30 
Calculation: (30 - 25) / 25 = .25 
Percent Improvement = 25% 

Participant B Pre-test Score: 18 out of 30 

Post-test Score: 30 out of 30 
Calculation: (30 - 18) / 18 = .67 
Percent Improvement = 67% 

Participant C Pre-test Score: 6 out of 30 

Post-test Score: 15 out of 30 
Calculation: (15 - 6) / 6 = 1.5 
Percent Improvement = 150% 



Table 2 



Company XYZ 

Training Effectiveness Report 



Name 
John Doe 
Jane Smith 
Fred Jones 



Percent Improvement 
25% 
67% 
125% 



Further Training Recommended 
No 
No 
Yes 



The Percent Improvement Method is limited in the information that it provides. The only 
conclusion that can truly be based on this report is that the participant scored better (or worse) 
on the post-test than he did on the pre-test. Hopefully, this is due to the training and subsequent 
learning that took place in the Basic Skills course. However, as Dr. Thomas Sticht points out in 
his report entitled, "Testing and Assessment in Adult Basic Education and English as a Second 
Language Programs" (1990), the increase in scores maybe due to other factors such as "warm- 
up," "surge," or "practice" effects. 
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Another problem in reporting learning gains in this manner is that it can be misleading, 
especially at a cursory glance. Taking a quick glance at Table 2, it would appear that Fred Jones 
had the best performance. (125% vs. 25% or o7%). However, careful consideration must be 
given to what the percent improvement represents. It is true that Fred had a greater increase 
in his assessment scores (pre vs. post); however, at the end of the course, he only was able to 
answer half of the assessment questions correctly. Both John and Jane received 100% on their 
final assessments. 

Three advantages that this method of evaluation has over other methods is its simplicity, 
objectivity, and confidentiality. For these reasons it has remained a popular method of 
evaluating learning. However, this type of evaluation fails to meet the needs of the training 
program in several areas. First, it does not show the areas of knowledge that the participant has 
mastered, nor does it prescribe a course of training. Also, the results lend itself to misinterpre- 
tation as the reader confuses Percent Improvement with Percent Correct. For these reasons, a 
second method of learning evaluation can be used to either enhance or replace the pre/post test 
method. 



Modules 

In the Modules method of learning evaluation, the general body of knowledge contained in the 
pre/post test is broken up into subsets or modules. Each module contains items that cover a 
specific area of knowledge. For example, a 50-question math assessment may be broken into 
five separate modules: 

1. Whole Numbers 4. Percents 

2. Fractions 5. Data Analysis 

3. Decimals 

Benchmarks are set that identify the number of correct responses needed to indicate mastery 
of each module. Instead of a generalized statement such as "40-hours of math training 
recommended," a more specific training program is recommended for each participant based 
on his/her performance in each module. This prescriptive method of evaluation also provides 
a basis for a more efficient training program. Time is not spent on concepts that the student has 
already mastered. The objective of the instructional program for each participant is to teach to 
the areas of deficiency so that the student can reach the benchmark levels that he/she previously 
failed to attain. Time is not wasted teaching and reviewing concepts that the participant already 
has mastered. 

As in the pre/post test method of evaluation, scores remain confidential. The company receives 
a report that indicates training recommendations for each participant. Table 3 on the next page 
provides an example of the educational provider's information obtained from a pre-test, 
followed by Table 4, a sample training recommendations report given to the company. 
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Table 3 



Educational Pr oviders Information 

(based on pre-test - confidential) 

Company XYZ 

Math Assessment — Benchmark Levels 

Whole Numbers 8 of 10 

Fractions 7 of 10 

Decimals 7 of 10 

Percents 7 of 10 

Data Analysis 7 of 10 



Educational Providers Information 

(based on pre-test - confidential) 



Participant 

John Doe 
Jane Smith 
Fred Jones 



Whole 
Numbers 
9 
8 
10 



Fractions Decimals Percents 



6 
4 
7 



8 
7 
8 



8 
5 
2 



Data 
Analysis 
5 
8 
3 



Table 4 

Company XY2 

Reporting Training Recommendations 

The training modules for each participant are recommended as follows: 

John Doe Fractions, Data Analysis 

Jane Smith Fractions, Percents 

Fred Jones Percents, Data Analysis 
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Another method used to report training recommendations/training effectiveness is through the 
use of individualized reports. Below is an example of a training recommendation report of Bill 
Taylor after completion of math training. 



Company XY2 

Shop Math Training Recommendations 

Name: Bill Taylor Date: March 23, 1994 



Modules v Further Training Recommended? 

Whole Numbers No 

Fractions Yes 

Decimals No 

Percents No 

Data Analysis Yes 



This method of evaluation is similar to the pre-/post-test method in that it utilizes one assessment 
tool to ascertain the participant's mastery of the material and allows for the confidentiality of 
the participant's score. It does, however, reveal those skill areas which the employee has and 
has not mastered. It is essential that the instructor create a positive rapport with the participants 
and reinforce the concept that the basic skills training is helpful — not punitive. 



In addition to the advantage of allowing for a more efficient use of training time and dollars, it 
also targets the instructional needs of each individual to prevent feelings of boredom or 
frustration that result from being placed in a class that is too easy or too difficult. Companies 
receive a better picture of their workforce's skills and can use this information as they plan other 
training projects. (For example, if 80% of the participants tested have been recommended for 
math training, the company may choose to delay Statistical Process Control classes.) 

The Module method of training evaluation is not without its drawbacks, however. Attempting 
to meet the needs of each individual by scheduling according to modules can become a logistical 
nightmare. In addition, instructors may find that they are providing individual tutoring within 
a whole class environment. Conducting this type of indiviuualized instruction does not provide 
many opportunities for group interaction and is riot the most efficient mode of training delivery. 



Competency Checklist 

As we continue to grow in our knowledge and delivery of workplace education programs, it has 
become evident that competency-based training is the most effective and efficient method of 
instruction. Competency-based (or performance-based) training is a systematic way of organiz- 
ing instruction to ensure that learners become competent in tasks that have been designated as 
essential for success on the job. 



Specifically, a competency-based curriculum includes the following (Stiles, Tibbetts, and 
Westby-Gibson, 1984): 
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+ Success is demonstrated by measurable knowledge gained rather than by time put in. 

+ The specified competencies and their standards for successful performance are carefully 
identified, verified, and communicated to all parties concerned. 

Instead of dealing with abstract concepts, competency-based instruction presents learning in 
real and concrete terms. It leads participants towards mastery of skills that are needed to 
function proficiently on the job. For these reasons, it has proven to be especially attractive to 
adults who may have unpleasant memories about their previous formal learning experiences. 

To design an effective competency-based workplace skills training program, two critical steps 
must be taken at its initiation: 1) conduct a literacy audit to identify the job tasks and the basic 
skills required for those jobs targeted and 2) determine, with the company training committee 
or company contact, a set of general competencies to include in the curriculum. In the literacy 
audit, identified jobs will be analyzed to break down the tasks performed into smaller units. 
These units will then be prioritized based on their importance for successful job performance. 
Based on this information, lessons are developed that provide instruction in these targeted 
skills. Conducting the audit, analyzing the information, and reporting the findings can be time- 
consuming and eat up a large chunk of funds budgeted for program development. The upfront 
cost of conducting a thorough literacy audit and task analysis is worth the expense. A good 
literacy audit will provide a solid foundation for the development of the workplace curriculum 
and is necessary for the design of an effective program. 

Competency-based instruction teaches to a pre-determined skill to be mastered. In this manner, 
competency-based instruction does u teach to the test." Lessons are developed that provide 
instruction and practice for those skills on which the participant will be tested. Similar, but 
never identical, practice exercises will be used in class. The mastery test determines the level 
of learning and transfer of the skill from the instructional material to the testing material. In this 
model, the competency-based testing takes place in the classroom setting. Taking this a step 
further, assessment can take place on-the-job to determine the mastery of skills that were 
identified as critical to proficient performance of a specific task. The proficient performance of 
the task and its components would indicate the transfer of the skill learned in the classroom 
setting to the actual workplace activity. 

For adults in a workplace setting, competency-based training is a means to provide skill specific 
instruction in a relatively short amount of time. The companies are satisfied because the 
instruction focuses on the employees' work skills, and the employees are enthused to see a 
concrete application of the lessons taught. 

The difficulty in conducting competency-based instruction and testing is the high degree of 
scheduling, recordkeeping, and reporting. Several issues of program design are: Should there 
be short modules to teach to a specific competency, or should a class focus on several 
competencies during a nine-week period? What about students who are in a nine-week course 
and have mastered several of the competencies that are being taught to this group? Do they 
receive the same instruction as the other students or should the instructor teach them 
competencies which they have not yet mastered? Should the instructor break the large group 
setting into smaller groups, or is one-on-one tutoring possible within the group? 
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Records to support mastery of specified competencies should also be maintained. It is suggested 
that an individual portfolio be kept on each student that contains all paper/pencil mastery tests, 
performance checklists, or anecdotal notes that back-up the assessor's rating of competency 
mastery. 

Each participant's evaluation report (or competency report) should be simple, yet informative. 
Over the last six months, we have experimented with several different formats for reporting 
competency mastery levels. The most effective and M user-friendly" report format that we have 
developed involves rating the participant's performance of the competency as Nonmastery (N), 
Partial Mastery (P) or Mastery (M). The instructor should go over each individual's performance 
and provide counseling on strengths and weaknesses. The company will also receive a 
competency report for each individual, with the mastery test scores remaining confidential. 
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Tools for Measuring Learning 




Pre/Post 

Assessment 

and 

Percent 
Improvement 



♦ Simplicity 
+ Objectivity 
+ Confidentiality 



Does not describe mas- 
tery 

Does not prescribe a 
course of further train- 
ing 

Increase maybe affected 
by "practice" effects 
Pre-test may discourage 
participants 

Percent improvement 
can be misleading 
Can encourage teaching 
to the test 



The company and partici- 
pants value the amount of 
learning 

Time is available for pre-and 
post-testing 

Adequate instruction time for 
measurable learning to take 
place 

Scores and percent improve- 
ment can be kept confiden- 
tial 

Participants are not discour- 
aged by testing 



Module cc mple- 
tion 



Effecient use of train- 
ing time 

Prescritpive recom- 
mendations for train- 
ing 

Confidential assess- 
ment scores 



Little opportunity for in- 
cluding other skills and/ 
or topics 

Management knows skill 
level of participants 
Difficult to schedule in- 
dividualized training 
plans 



Specific training outcomes 
are defined 

Assessments and curriculum 
arc available for each out- 
come 

It is possible to schedule in- 
dividualized learning plans 
It is agreed that management 
can have access to each em- 
ployees' skill level 



Competency 
Checklists 



+ Learning presented 
in real and concrete 
terms 

+ Job-related 

+ Focused instruction 

+ Individualized re- 
ports 



High degree of record- 
keeping and reporting 
Require effective and 
valid measurements for 
each competency 



♦ 
♦ 



Competencies have been ar- 
ticulated and agreed upon 
Assessment measures exist 
Opportunity for additional 
training to improve mastery 
is available 
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Project recommendations: 

For the measurement of learning we recommend the following: 

1 . Module completion is practical only when there is a concise and well-defined list of skills 
to be taught and it is possible to utilize individualized scheduling. The only basic skills 
content area with which we have been able to use module completion effectively is math. 

2. Competency checklists are preferable to pre/post assessment because of the reasons 
indicated earlier. A post-assessment can be used, as well as portfolios and task 
performance, to document the achievement of the competencies. (See Exhibit 5 for a 
sample of a competency checklist.) 

3. Because of the value of measuring basic skills as they apply to specific work situations, 
we recommend customized assessments. (See Exhibit 6 for an excerpt from a custom- 
ized assessment.) 



L Mrowlckl / J. Conrath — 



34 



The Center - Resources for Education and Northern Illinois University - Business and Industry Services 

- 24 - 



Exhibit 5 



Workplace Communication Competencies 
February, 1994 — June, 1994 



Name: 



Date:. 



iV — Nonmaslery of Competency 
P — Partial Mastery of Competency 
M — Mastery of Competency 



ORAL COMPETENCIES 

I. Describes job responsibilities at 
concise manner. 



in a clear, 



2. States the correct at jreviations used for various 

products and procedures. 

3. Describes his/her production process. 

4. Relates various problems which may arise during machine operation. 

5. Describes corrective actions. 

6. Describes the work order — including all abbreviations, types of 
steel, pounds required, skids required, when customer needs order, etc. 

7. Describes finished product and what happens to finished product 

at end run. 

8. Lists the attributes of the steel used at his machine (it. type, 
gage, finish, dyes and weight). 

9. States preventative maintenance. 

10. Describes how the thickness of steel is measured. 

1 1 . Describes math concepts used in steel production at 

12. Names the various departments. 

13. Names the supervisors and managers of each department. 
|4. Stales the location of various stock. 
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BEST COPY AVAILABLE 



15 



Names the functions of the various departments at 



16 Describes the importance of teamwork. 

17. Relates the importance of quality. 

18 Describes the general goal of ISO 9000. 

19. Defines various TQM terminology such as: 



a. 


internal customer 


b. 


external customer 


c. 


supplier 


d. 


total quality management 


e. 


commitment to quality 


f. 


teamwork 


fi- 


customer satisfac* : on 


ll. 


flow charting 


i. 


excellence in productivity 


Explains various 


a. 


vacation 


b. 


sick leave 


c. 


leave of absence 


d. 


tardiness 


e. 


overtime 



policies: 



21. Explains the importance of flow charts, bar graphs, line graphs 
and pie graphs. 



READING AND WRITING COMPETENCIES 

22. Reads and comprehends textbook materials and various handouts 



23. Answers, in short concise sentences, questions related to text, 
handouts and materials. 

24. Uses a dictionary to define workplace vocabulary. 

25. Constructs a graph on depicting his/her monthly household expenses. 

26. Reads and comprehends various TQM literature related to graphs and 
flow charting. 

27. Writes job responsibilities in a sequential manner. 

28. Fills out Maintenance Request forms. 

29. Reads and interprets work order forms. 

30. Reads, comprehends and writes alt abbreviations used in 



(independent and 



handouts) presented in class. 



production processes. 
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[V. Answer the following questions. 



7. What is the name of the department for the job schedule shown below? 



8. Which jobs are scheduled to ship on 8/6/93? (List by job number.) 



JOB SCEEDQLE 00-E8GIH 01-SHGIK. BSPEATS 05-OOTSIDE SHEARING 
10 -SHEAS 15-POHCH (5-FQRH 

08/12/93 50-SPOTHRLD 55-KELD T5-PAI8T 

100-SaiPFIHG 999-BOLDIKG BETXEEH EHG k SfiEAB 











SHIP 


IIS 


EHG 


DEPT. 


JOBS 


C0STQHER 


DESCRIPTION 


DATE 


CHG 


COHHEHTS DOE 


SO 


3563 


BLANKET 


2 BB1040 RACK H/CASTE8S 


06/24/93 


:::: 


JOB (5(36 


50 


2890 


BLANKET 


BISC PROJECTOR CAilS 


06/25/93 




FOR JOB 15(37 


50 


3563 


BLAHKEI 


1 BH1040 RACE S/CASTIBS 


07/02/93 




JOB 15(32 


50 


53(6 


BLAKKKT"" 


2( SETS SD (8 I 76-1/4 


07/02/93 




FOR STOCE 


50 


5(47 


blahie; 


LBO 10 I 90 OFBIGHTS 


07/09/93 






50 


3564 


OS PS 


1 H-l-78 ACT TAG BACK 


07/16/93 




JOB »55(6 


50 


3563 


BLAHEET 


1 H-4-91 ACT TAG BACK 


07/19/93 




JOB 15430 


50 


5512 


DESES 


5 ES22984 B/5 SHELTES 


07/21/93 




BLAHEET f 3454 


50 


3<5< 


BLAHEET 


7-ES22984 CABINETS 


07/21/93 




7 HEEDED JOB 15512 4 5634 


50 


3564 


DSPS 


1 F-2-78 ACT TAG EACE 


07/31/93 




JOB 15573 


50 


5638 


BLANEET 


100 BOTTOM SHELVES K/RKTAI808/04/93 






50 ' 


5(57 


OS PS 


6 BR-1040 K/CASTSBS 


08/04/93 




BLAHEET 15637 


50 


5521 


OFFICE IHT 


5 RDBC363015 BOOECASE 


08/06/93 




CHARGED TO SDBC ISS0ED HEX JOB 


50 


5669 


BLAHEET 


29 SETS REPLACE H8SE0K DBS 08/06/93 




SEHT O0T 8/6 FOR POHDER COATIHG 


50 


5(79 


H0DE3N OF 


11 SETS SD 36188- 1/4 V/STO 08/09/93 




BLAHEET 15465 


50 


5663 


ATLAHTIC B0S2 8(116 RKCTCLIBG SOUTHS 


08/13/93 




BLAHEET #3737 


50 


5580 


BLAHEET 


10 BDH186425 


08/13/93 






50 


5214 


CRET08S 


30- 4-XEEEL XAGOH FRAME 


08/16/93 






50 


3563 


BLAHEET 


2 H-4-91 ACT TAG BACI 


08/20/93 




JOB 15560 * 2 B81010 


50 


5550 


DSPS 


2 BR1040 HO CASTEBS 


08/20/93 




BLAHEET f 5637 X/K-4-91 


50 


5(90 


HODRRH OFF 


8SD 30I100-1/4A1-36I100-1/408/20/93 






50 


2926 


BLAHEET 


110 1200 BOOICASES 


08/20/93 




(5 HEEDED 


50 


5519 


0HITED BOS 


1 SET SD 36 I 64-1/4 


08/20/93 






50 


5690 


STORAGE 


4 SETS 48 I 88-1/4 


08/20/93 




BLAHEET 15320 


50 


5572 


DSPS 


1 BR1040 HO CASTEBS 


08/20/93 




BLAHEET *5637 


50 


5684 


BRADFOSD 


1 SET SD 42 I 85-1/4 


08/20/93 




BLAHEET 14865 


50 


569 1 


SPACES A?KB 


4 SETS SD 42 I 88-1/4 


08/20/93 




BLAHEET 1(6(6 


50 


5575 


STARE 


1 8(2118 BICICLIKG .S08TKR 


.08/23793 




Bimtt. 13797 



4 
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VIII. 



Daily Job/Cost Card 



14. 



You make a set-up from 7:00 AM to 7:40 AM, you run production from 
7:40 AM to 8:50 AM, and you do rework from 8:50 AM to 9:00 AM. 
"ttv** Is -for doi>1f Sooo. 

Fill out the Viking Daily Job/Cost Card completely, indicating the 
following: 



a. 
b. 
c. 
d. 
e. 
f. 



Employee number 



Job $ 

Dept. # (FUw^ear d^.#)VIKING daily job/cost card 
Operation # 
Employee # 
Set-up time 
Run time 
Rework time 



Xit OA CI U»*OA« 



tlOV<*<4 W3 



i oa r*oouc i 

S«P 



rui oa coca *o 



J ©a El 



»*mw. oa co**"vnc 
Po.C 



OtMtnrrr counr Co 



Yo.N 



Y« N 



JO*OA*m>*4CI i>*OA« iOI 4COXMU *o r a*1 OA COOt fcO 



So.P 



WO^ C<«T<*« 




















J o« E 





Po-C 



Yo-N 



X>«OAMO>A(C1 LA*OA« U3T l(OU(t« *0 f AA1 OA coot »o 



Jdu* oa Mcoucno** 

Sc*P 
oo*.»na> cownruo 



■ 




* CO»<uAAC#nx>*»OACW#vonCI 


t r*<noA 






*OA«Q*«C 0* .*>** 








J o. E 





paktm*. OA cou*vc7C 



PO-OA* 

Yo,N 



T0OUM3 

Ye-N 



8 

BEST COPY AVAILABLE 
38 



Level 3: Transfer of Learning 



The purpose of the third level is to measure the degree of application of the new skills to the 
job. While virtually every program measures the amount of learning at the end of the course, 
few attempt to measure how much of that learning is being utilized on the job. 

This level of evaluation offers the program the opportunity to demonstrate its value to the 
organization. While providing data on the job, it can also be a necessary link to measuring the 
impact on the organization. By showing a positive effect on the job, this level of evaluation can 
also facilitate the support of supervisors and managers in continuing the program. 

There are several difficulties in successfully measuring this level. In the ideal situation, specific 
documentation regarding the application of basic skills is needed before the training as well as 
after the training. If the purpose of the course is to provide workers with the reading skills 
necessary to comprehend the written job aids, the program should be able to note and record 
how often the job aids are accessed prior to the training. After the training, data should be 
collected regarding how often the job aids are then accessed and comparisons made of the 
difference before and after the training. Ideally, it would be possible to measure and compare 
"the before" with "the after." 

Realistically, such data is not available to collect because it is time-consuming and possibly 
disruptive to the flow of work. It also places value on certain tasks which may not in actuality 
be valued. For example, if a worker needs certain information regarding how to operate a 
machine, he or she can access a job aid or a co-worker. While the program may emphasize the 
importance of accessing the job aid, in actuality accessing a knowledgeable co-worker may be 
more time-efficient as well as more effective. Therefore, the ability to access the job aid may 
be less important than the ability to access the co-worker. 

Performance checklists are one tool for recording usage of skills on the job. A performance 
checklist lists the steps which are carried out to perform a task. An observer then visits the 
worker, notes the task being performed, and completes the checklists. All checklists are then 
analyzed to determine how well the skills are being applied. 

If such data is not available for collection and analysis, the next best step is to measure the 
"perception" of the transfer of learning. This can be the perception of the workers themselves 
as well as other people involved such as lead workers, team workers, supervisors, or managers. 

One tool is the survey which can be given to supervisors to complete about the workers who 
participated in the training. While this tool can be subjective in nature, it is a very quick and 
convenient method for obtaining feedback on performance. One cautionary note is in order 
for using this tool. If the program is voluntary and if the workers know that the supervisor will 
be more closely examining their work, the use of surveys may act as a "dis-incentive" to 
participation in the program. 
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Another tool is the interview. An interview or a focus group of supervisors can provide 
feedback on the transfer of skills. Interviews are more open than surveys although they, by their 
very nature, require more supervisors' time and more program time to analyze and compile the 
results. 

Both the survey and the interview can also be used as tools for self-appraisal. Workers 
themselves should be given the opportunity to reflect on the application of the training. Such 
realistic feedback can also be very useful in validating or revising the curriculum. 

The project has found that the most promising tool for transferability of skills is the perfor- 
mance indicator rating instrument. Such an instrument is customized developed by the 
supervisors themselves. A facilitator guides the supervisors in identifying indicators of perfor- 
mance. The next step is to develop observable measures typical of highly competent, average, 
and less-than competent workers. A 3- or 5-point rating instrument is developed which 
supervisors then complete. The assumption behind the performance indicators is that basic 
skills in an integral part of the performance. 

Performance appraisal ratings can be another tool. The major drawback to the use of these 
instruments is that they are usually completed annually and usually at the anniversary date of 
employment. Neither may be appropriate in terms of the participation in or conclusion of the 
training. 

The following chart summarizes tl^r strengths and weaknesses of each tool. 
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Tools for Transfer of Basic Skills 




Performance 
checklists 



+ Objective 
♦ Job-related 



♦ Requires a solid job task/ 
basic skills upon which 
to develop the checklist 

+ May be difficult to 
develop 

♦ Likely to require a skilled 
observer to administer 

♦ May require extended 
time to administer 

♦ May be disruptive to the 
workflow 



+ Performance checklists are a 
part of the existing supervi- 
sion system 

♦ Skilled staff are available to 
develop and administer 

♦ Workers and other staff are 
agreeable to their use 



Could dis-incent some 
workers from participat- 
ing in the program 



Participant 
questionnaires 



+ Can provide anony- ♦ Subjective 
mous feedback 

+ East to collect data 

♦ Can be customized 



♦ Workers have the basic skills 
to complete the question- 
naires 

♦ Completion is regular and 
consistent 



♦ Responses arc easy to 
compile and analyze 



Participant ♦ Encourages wide 

interviews variety of responses 



♦ Subjective 

♦ Requires time to sched- 
ule 

+ Requires a skilled facili- 
tator and reporter 

♦ Some staff may not be 
willing to openly share 
perceptions 



+ Time is available 

♦ A skilled facilitator and 
reporter is available 

♦ Staff are positive about shar- 
ing perceptions 

♦ An interpreter is available if 
needed 



7 



ft 
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Tools for Transfer of Basic Skills 

(continued) 




Coworker, 
supervisor, 
manager 
questionnaires 



+ Can allow for anony- 
mous feedback 

♦ Easy to collect data 

+ May encourage 
program support 

♦ Can be completed 
during non-produc- 
tion time 



♦ Subjective 

♦ Requires that everyone 
understand the purpose 
of the program and the 
relationship to the job 

+ Requires time to 
complete 

+ Could dis-incent work- 
ers from participating in 
the program 



+ Completion is regular and 
consistent 

+ Responses are easy to com- 
pile and analyze 



Co-worker, 
supervisor, 
manager 
interviews 



+ May encourage 4- Subjective 



program support 

Encourages wide 
variety of responses 



♦ Requires time to sched- 
ule 

♦ Requires a skilled facili- 
tator and reporter 

♦ Some staff may not be 
willing to openly share 
perceptions 



4- Time is available 

♦ A skilled facilitator and re- 
porter is available 

4- Staff arc positive about shar- 
ing perceptions 



Performance 
indicator rating 
instruments 



+ Developed by the 
company staff 

+ Job-related 



4- Requires time to develop 4- Time is available 



4- Requires a skills facilita- 
tor 



4- A skilled facilitator is avail- 
able 



Performance 
appraisals 



4- Arc an established 
part of the system 



4- Staff can be trained to ob- 
serve and fill out the instru- 
ment consistently 



Needs to include basic 
skills in the appraisal pro- 



cess 



♦ Supervisors must be 
trained to observe and 
note use of basic skills 

4- Infrequent usage 



4- Effective, consistent ap- 
praisal system is in existence 

4- People completing the ap- 
praisals have the knowledge 
to rate basic skills 



ERJC 
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Training Impact Survey 



Employee's Name: Class: 

Supervisor's Name: Date: 



Please evaluate the employee's on-the-job skills and performance as a result of his/her recent 

participation in a Basic Skills class. Return this completed form to 

by /Thank you for your cooperation. 



Work Competencies 


Yes 


No 


Not 
Sure 


Not 
Applicable 


1. 


Has the employee's production increased? 










2. 


Has the employee's scrap or rework been reduced? 










3. 


Has the employee been involved in an accident/injury 
during this training period? 










4. 


Has the employee's self-confidence noticeably 
increased? 










5. 


Has the quality of the employee's work improved? 










6. 


Was the employee promoted during or after this 
training period? 










7. 


Has the employee become more effective in teamwork 
roles? 










8. 


Has the employee's participation in staff/department 
meetings increased? 










9. 


Docs the employee exhibit a more positive attitude 
about the company and his/her job? 










10. 


Has the number of communication problems with 
the employee decreased? 










11. 


Docs the employee speak English more often to you, 
other supervisory personnel and/or co-workers? 










12. 


Has the employee's writing skills improved? 










13. 


Has the employee's computation and measurement 
skills improved? 
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Exhibit 8 



Sample Performance Indicators Developed By Supervisors 

The following is an excerpt of performance indicators developed by supervisors at one 
company. The. facilitator of these indicators was Jorie Philippi of Performance Plus Consultants, 
Inc. Listed below is a summary of actual comments from Focus Group Discussions at one 
company. 



n, 9u fl1< tY^ onsciou*nc88 



4 2 



5 

Talks about specific 
measurements and uses , 
quality terms 


3 

Expresses concern about 
doing a Job the right way 


1 

Makes many errors on the 
Job 


Doesn't rush through a Job or 
take short cuts 


Uses correct kinds of math 
when needed for the 
situation 


Passes on bad parts without 
checking or correcting them 
or telling anyone 


Checks work thoroughly 
throughout process to ensure 
accuracy 


Performs procedure to the 
best of his/her ability 


Makes errors in simple 
math or" record keeping 


Figures out the cause of a 
problem 




Cannot use quality terms or 
data 


Has a low reject rate 




Is unaware of the right 
information to ask for 


m. Working with Others 
4 2 


Pitches in to help others with 
problems 


|3 

Does what he Is supposed to 
do 


ii 

Is unable to adjust to other 
co-workers 


Offers to help others when 
done with own Job 


Shows an open-minded 
attitude 


Complains constantly 


Listens to everyone's concerns 


Displays cooperaUve 
behaviors when in a group 


Argues 


Shows respect for opinions of 
others 




Seldom speaks to others 


Makes helpful 
suggestions 




Jumps to conclusions 
quickly 


Is aware of differences and 
tries to work out good 
compromises 




Sees only one side of any 
problem 






Blames others for problems 






Refuses to help others unless 
told he/she has to 
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Project recommendations: 

The project recommends the following: 

1. Administering and compiling the results of supervisor surveys is valuable to the program. 
Besides giving feedback on the skills transfer, it actively involves the supervisors in the 
program. (See Exhibit 7 for a sample survey). 

2. If time permits, development of performance indicator rating instruments is the most 
valuable tool. (See Exhibit 8). 

3. Participants' own perceptions of the effect of the training is also important. Their feedback 
can be elicited during the training as well as after the training. 
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Level 4: Organizational Impact 



The fourth level of evaluation addresses the issue of organizational impact. The fundamental 
question that is raised is, "How well is the basic skill program adding value to the strategic 
direction of the organization?" This is not an easy question to answer. The degree of difficulty, 
however, should not inhibit the pursuit of an agreed upon analysis of impact. 

Company-wide basic skill programs have two major operational challenges. The first is to 
provide the strategic planning, mobilize the human and financial resources, gain the commit- 
ment and communicate the vision to begin the basic skills initiative. The second great challenge 
is to sustain the effort over a long period of time. It is to this second challenge that the 
measurement of organizational impact has a direct line. In our experiences, first, second and 
third level evaluation data are enough to sustain a basic skills program for the first year or so. 
Positive employee feedback, a boost in morale, positive results from pre/post-assessments, 
competency checklists, and verified job relevance may be enough. Combine that with some sort 
of subsidy to assist with the cost of training and you have a formula for success in the initial stage 
of development. Longevity will, in large part, depend upon the degree to which the basic skills 
program can be linked to organizational strategic initiatives. 

Cost Benefit Analysis 

Many companies want to know if the basic skills program is having any impact on the bottom 
line. To answer this question in financial terms requires that a cost-benefit analysis be 
performed. A calculation is made that takes into consideration the amount of money that 
improved employee performance saves or generates for the company. In order to make the 
computation, pre/post data must be available. In addition, there may be a need to establish a 
control group or site that could construct a baseline. Typical production performance 
indicators are listed in Table 1 . Additional cost-benefit analysis information could be calculated 
by determining the costs that would have been made had the basic skills program not been in 
place. Conducting a cost-benefit analysis can be a good thing to do especially if there is a high 
correlation to the performance measure. The problem is that most continuous quality 
improvement initiatives do not improve because of a singular reason. For example, scrap rates 
are usually reduced because a more efficient process was put into place or new equipment was 
purchased. Effective training on the new equipment or process was a critical element to scrap 
reduction and basic skills was a critical prerequisite to the success of the training. 

Linking Basic Skills To Strategic Initiatives 

Establishing the linkage between the basic skills program and the company's strategic direction 
must be established early. Up-front planning should occur that takes into account the "big 
picture." What is happening in the organization? Are there external forces that are having an 
impact on the business? What continuous improvement efforts are going on and how does a 
basic skills program fit in? The key to measuring organizational impact is: 1) tying the program 
to measures, and 2) ensuring that ail levels of evaluation are linked in a logical chain. 

Organizations that take into account strategic issues and link those issues with measures and 
behaviors get off to a much better start than those that do not. There is always the concern over 
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a basic skills program that starts because there is a grant or because the competition is doing 
it. The value is neither established nor is it linked to any measures or behavior changes. 

An example of a company getting off to a good start would be one that is embarking on a Total 
Quality effort. Teams whose main focus is reducing quality failure costs are deployed. One of 
the recommendations from the teams is to implement a statistical process control program. The 
skill requirements versus the current skill levels indicate a gap. A basic skills program is 
implemented that focuses on math skills and writing for corrective action. The basic skills 
initiative is linked to a larger quality issue. If quality failure costs in this example are reduced, 
then this is a direct link between the cost savings and the basic skills program. If not for the 
success of the SPC program, failure costs would not have been reduced. If not for the employees 
monitoring the system and maintaining control charts, SPC could not have been implemented. 
If not for the basic skills instruction, control charts could not have been maintained. 

The rules of evidence for establishing organizational impact will be different in every company. 
What matters most is that the linkage of basic skills to strategic issues is clear and agreed upon. 
Measuring impact inside companies is not the same as convincing the scientific community. Is 
it important that you attribute 15% of a cost of quality reduction to a basic skills initiative? No, 
but it is important to establish that the basic skills program was a necessary component of a 
larger training strategy. There should be a job analysis that clearly shows where employees need 
to perform and where they are performing now. Basic skills instruction is one strategy to fill the 
gap. If the linkage is clear, then it is our position that the percentage of improvement is not 
important. 

Another recommendation is to tie the basic skills program to the actual performance unit or 
closest to the point of implementation. Using the SPC example mentioned above, a company 
increase in net profits attributed to the basic skills program would be too diffuse. Cost savings 
in the Fabrication Department where the SPC program is located, would be more appropriate 
and convincing. 

There are several performance indicators that can be considered when linking basic skill efforts 
to company continuous improvement initiatives. As mentioned previously, increases in 
financial/market performance (sales growth, profitability, market share) for most companies 
would not be a compelling linkage. Strategic planning goals and objectives tied to customer or 
industry recognition are improvements in operation and probably more appropriate and easier 
to agree upon. Table 1 is a list of possible performance indicators and measures. 
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Table 1 



Performance Indicators and Measures 

1 . Customer or Industry Recognition 
Customer-Supplier Award or Recognition 

♦ Ford Ql 

+ GM Targets for Excellence 

ISO 9000 Certification Industry Quality Awards 

♦ Malcolm Baldrige 

Customer Satisfaction 

+ surveys 

+ letters 

+ other forms 

2. Operational Improve «*mts 
Customer Delivery 

+ Reductions in order-to-delivery lead times 
+ On-time delivery rates 

Production Efficiency 

♦ Labor productivity (units produced per employee or labor hour) 

♦ Reductions in changeover or setup time 
+ Capacity utilization 

4- Value-added per employee 

♦ Efficiency (actual vs. planned) in launching of new product lines 

♦ Labor utilization 

Costs of Quality 

♦ Scrap rates 

+ Customer returns 
+ Rework costs 

Maintenance 

♦ Reductions in unscheduled down time 

+ Reductions in response and repair times and costs 

Material Management and Control 

+ Reduction in work-in-process inventories 

+ Reductions in error rates in production of company reports 

Employment Costs 

♦ Employee turnover, attendance/absenteeism 

♦ Workers' compensation costs 

Sales and Customer Service 

+ Average length of order-to-delivery lead times 

♦ Average variance between quoted and actual delivery dates 
+ Percentage of orders not meeting quoted delivery date 

+ Percentage change in sales of product, not lines, etc. 

L. Mrowicki / J. Conrath - The Center - Resources for Education and Northern Illinois University - Business and Industry Services 

- 32 - 

48 



Tools for Measuring Impact 




Cost-benefit 
analysis 



♦ Quantifiable 

♦ Commonlyused tool 
in training 



♦ Difficult to isolate costs 

♦ Difficult to provide a cost 
figure to the benefits 

+ Requires quantifiable 
data which can be 
tracked to the cost as 
well as the program 

+ Dependent on no other 
factors affecting the out- 
comes 



♦ The cost of a lack of basic 
skills can be determined 

+ The services provided by the 
program can be costed 

♦ The impact of the program 
can be documented and 
costed 

♦ No other factors have influ- 
enced the impact 



Linking basic 
skills to strategic 
initiatives 



+ Basic skills programs 
arc recognized as an 
integral part of the 
company 

♦ Relevant to TQM 

+ Utilize current cost 
of quality data 

+ Variety of perfor- 
mance indicators 



Docs not attribute a spe- 
cific monetary benefit 
from the program 

Dependent upon well- 
articulated strategic 
plans 



♦ The company has imple- 
mented a TQM initiative 

+ The company has established 
a cost of quality and regu- 
larly measures it 

♦ The data is available for analy- 
sis 
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Project recommendations: 

For the measurement of impact, we recommend the following: 

1 . In our experience with this project, cost-benefit analysis has not proven to be reliable for 
the following reasons. First, it is difficult to establish a cost for the lack of basic skills, as 
well as to establish a cost for the program services, especially when the services are 
partially grant-supported. Second, many factors contribute to the cost of problems. 
Third, cost data is difficult to isolate or, because of its proprietary nature, it is not 
available. Fourth, data may not be available as to individual participants' performance. 
(This is especially true in work-team environments and when participation in the 
program has been voluntary.) 

2. Valuing basic skills according to strategic initiatives offers the greatest promise of 
measuring impact. Utilizing costs of quality data in those companies who are beyond the 
first stage of TQM adoption and tracking the value of the basic skills program can lead 
to validation of the importance of basic skills programs. This is a promising practice 
which we highly recommend for field-testing and one which we will be using in future 
projects. (See Exhibit 9.) 
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Exhibit 9 | 



Example of a Performance Indicator Report 

ABC Corporation is a medium-sized manufacturer. The company has been rapidly changing 
over the last five years. The average order size, lead time, and profit margin is half of what it 
used to be. ABC developed two strategies to improve the company's performance. 

The first phase of ABC's strategy was to ensure that the plant and equipment would match the 
needs of their customers. ABC invested in high speed processing equipment to guarantee quick 
response to customer needs. 

The second phase of ABC's strategy was to fully utilize their greatest untapped asset, their 
employees. ABC needed all its employees to understand how these strategies affected their 
areas of responsibility and what changes would be needed to capitalize on them. To strengthen 
their ability to react, they needed a better educated, more flexible workforce. The best catalyst 
for this strategy would be the TQM process. 

In September of 1992, ABC started on the unending process of continuous improvement. To 
be successful, they felt they needed a program that was fully integrated for each area of the 
business. An aggressive schedule to train all employees in how the business intended to operate 
was established. 

The program assessed and then provided training to bring each department of the organization 
to the desired skill levels to ensure that the TQM process would be successful. Like many 
manufacturing companies in Illinois, ABC is a melting pot of cultures and languages, which 
meant that extensive twining of virtually every employee would be necessary. Employees had 
the difficult task of competing both their regular job duties and attending the training program. 
To eliminate objections to the training, union employees participated in training either before 
or after their regular shifts in paid overtime sessions. 

The training plan included these basic skills courses: 

♦ Basic Shop Math 

♦ Reading 

♦ English as a Second Language 

The basic skills courses played an integral role in developing employees' skills to participate in 
further training such as Quality Management Skills training, SPC, and Team Leader Training. 

Both the basic skills and the technical training took place over 18 months and resulted in over 
265 hours of training representing over 8,000 man hours. A tutor/mentor program has been 
developed to augment the classroom training to provide additional customized, one-on-one 
training. 
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One clear focus resulted in both the intangible and tangible benefits for both the company and 
the employees. Some of the intangible benefits are: 

♦ Greater employee participation/involvement. 

+ Greater cross-functional understanding and cooperation. 

+ Improved management skills. 

+ Improved communication. 

+ Focus on customer satisfaction. 

+ Higher level of education. 

+ Better and increased decision-making capability. 

+ Ability to empower workers. 

+ Employees taking more responsibility. 

+ More initiative. 

+ Higher level of trust. 

+ Increased job satisfaction and fun. 

+ Improved housekeeping. 

+ Transferable skills. 

Specific company-wide performance indicators include: 

+ Improved overall product quality. 

+ Increased on-time delivery performance to 95%. 

+ Increased inventory accuracy, approaching 99-5%. 

+ Lowered internal scrap and handling damage 25%. 

+ Sales have increased dramatically. 

+ Increased safety, over one year with no lost-time accidents. 

+ Decreased Workers' Compensation Insurance premiums. 

4* Numerous vendor of the year or quality awards by customers. 

The basic skills program was deemed an essential component to achieving this initiative and an 
important contributor to improving the company's performance. 



52 



Bibliography 



I 



Carnevale, A., Gainer, L. & Metzer, A. (1990). Workplace Basic Skills Training Manual. San 
Francisco, CA: Jossey-Bass. 



Kutner, M. Sherman, R., & Webb, L. "Review of the National Workplace Literacy Program. " 
(1990). Washington, D.C.: U.S. Department of Education. 



Manly, Donna. Workplace Education Checklist: A Tool for Assessing And Improving 
Performance. (1994). Madison, WI: Center on Education and Work - University of 
Wisconsin. 

Mikulecky, L. and Lloyd, P. (1992). Evaluating the Impact of Workplace Literacy Programs. 
Bloomington, IN: Indiana University. 

Philippi, Jorie W. "How Do You Know If It's Working? Evaluating the Effectiveness of 
Workplace Literacy Programs" (1992) Springfield, VA: U.S. Department of Education. 



Philippi, Jorie W. Literacy At Work: The Workbook for Program Developers. (1991). 
New York: Simon & Schuster, Inc. 



Rothwell, Dr. William J. and Brandenburg, Dr. Dale C. The Workplace Literacy Primer - An 
Action Manual for Training and Development Professionals. (1990). Amherst, MA: 
HRD Press. 



Sticht, Dr. Thomas G. " Evaluating National Workplace Literacy Programs. "(1991). El Cajon, 
CA: Applied Behavioral & Cognitive Sciences, Inc. 



Sticht, Dr. Thomas G. "Testing and Assessment in Adult Basic Education and English as a 
Second Language." (1990). Washington D.C.: U.S. Department of Education. 



Stiles, R., Tibbets, J., and Westby-Gibson, D. (1984). "Why CBAE?" Unpublished manuscript: 
Center of Adult Education - San Francisco State University. 



The Bottom Line: Basic Skills in the Workplace. (1988). Washington, DC: US. Departments 
of Education and Labor. 



L. Mrowickl / J. Conrath - The Center - Resources for Education and Northern Illinois University - Business and Industry Services 

35 



53 



